1. Field of the Invention
The present invention relates to an image signal processing apparatus, an imaging apparatus, an image signal processing method and a computer program, and more particularly to an image signal processing apparatus, an imaging apparatus, an image signal processing method and a computer program for performing signal processing of imaged data by a solid state imaging device of a single plate color system.
2. Description of Related Art
A general solid state imaging device of the single plate color system has a color filter stuck thereto to transmit a specific wavelength component in each pixel to a surface of an imaging device, and restores necessary color components by a set of a plurality of pixels. At this time, for example, a Bayer color array that expresses red (R), green (G) and blue (B) by a set of four pixels as shown in FIG. 1 or the like is used as the color array used for the color filter. Because each pixel has only the information of a single color component like this in the solid state imaging device of the single plate color system, demosaic processing that restores necessary color components in each pixel by performing interpolation processing using the color information of surrounding pixels is performed.
The configuration of an imaging apparatus equipped with a solid state imaging device of the single plate color system is shown in FIG. 2. A solid state imaging device 13 of the single plate color system receives the light that transmits a color filter 12 among the incident light through an optical lens 11. An image signal that is photoelectrically converted by the solid state imaging device 13 to be output as an electric signal is converted into a digital signal by a not shown A/D converter. After that, the converted image signal receives clipping processing, gamma correction, white balance correction, demosaic processing and the like in a camera signal processing unit 14, and the processed signal is transmitted to an image compressing unit 15. The image compressing unit 15 reduces the amount of data of the image signal, and converts the reduced image signal into a predetermined recording image format to output the converted image signal. A recording unit 16 records the converted image data on a recording medium. Hereupon, it is not always necessary to perform the image compressing processing, the image compression is ordinarily performed because the number of pixels of an imaging device has increased in recent years and the miniaturization of an apparatus itself has been required.
With reference to FIG. 3, the demosaic processing of an image obtained by the solid state imaging device of a single plate color system is described. The solid state imaging device of a single plate color system is configured to perform imaging through a color filter having a color array such as the Bayer color array of the primary color system (see FIG. 1) or the like, and is configured to obtain only the signals having a specific wavelength to each pixel, i.e., the color component data of a specific wavelength. In a case of using the solid state imaging device of the single plate color system having the Bayer color array, an output image 20 of the solid state imaging device becomes a color mosaic image having only one piece of information of R, G and B at each pixel.
A demosaic processing unit 21 executes the processing of restoring all pieces of information of each color component data, i.e., R, G and B, by performing color interpolation processing to each pixel.
First, the restoration of a G signal which restoration is executed by the demosaic processing unit 21 is described. In the Bayer color array, the G signal is obtained in a checkered pattern. At a pixel at which no G signal exists in the output image 20 of the solid state imaging device, for example, a case of G11, the G signal is generated by interpolation processing based on surrounding G signals. To put it concretely, the G signal (G11) is restored in accordance with the following expression.G11=(¼)(G01+G21+G10+G12)
Next, the restorations of an R signal and a B signal are described. In the Bayer color array, the data of both of the R and B exist every other pixel line. For example, R signals exist but no B signals exist in the pixel line of the top rung of the output image 20 of the solid state imaging device shown in FIG. 3. Moreover, B signals exist but no R signals exist in the second pixel line.
In a pixel line in which either data R or data B exists, the data R or the data B is obtained every other pixel. In the case where an R signal (B signal) exists in the same line as that of a pixel at which a certain R signal (B signal) does not exist in the output image 20 of the solid state imaging device, for example, cases of R01 and B12, interpolated pixel values in the pixels in which the R and B signals do not exist on the pixel line can be calculated by the following expressions, and the R signal (B signal) of each pixel can be restored.R01=(½)(R00+R02)B12=(½)(B11+B13)
In the case where R signals (B signals) exist in the same column, for example, cases of R10 and B21, the interpolated pixel values at the pixels where certain R and B signals do not exist can be similarly calculated in accordance with the following expressions, and the R signal (B signal) in each pixel is restored.R10=(½)(R00+R20)B21=(½)(B11+B31)
Moreover, in a case where no R signals (B signals) exist in both of the same line and the same column, for example, cases of R11 and B22, the interpolated pixel values in the pixels in which certain R and B signals exist can be calculated by the following expressions, and the R signal (B signal) at each pixel is restored.R11=(¼)(R00+R02+R20+R22)B22=(¼)(B11+B13+B31+B33)
The demosaic processing unit 21 performs the color interpolation processing as mentioned above, and outputs R signals 22r, G signals 22g and B signals 22b to all pixels. It is noted that the above interpolation processing is only one example, and any color interpolation processing using the correlations with the other color signals may be performed.
Various image processing methods for reducing noise to improve the image quality of an imaging apparatus equipped with a solid state imaging device of a single plate color system have been proposed. For example, Georg Petschnigg Et al, “Digital Photography with Flash and No-Flash Image pairs,” acm Transaction on Graphics, Vol. 23, Number 3, pp. 664-672, August 2004 (a non-patent document 1) discloses a technique of photographing a plurality of images having different spectral characteristics one after another and utilizing the plurality of images having the different spectral characteristics to obtain an image in which noise is reduced. That is, the technique photographs the following two kinds of images sequentially:
(a) images that have been photographed with spectral characteristics near to a target and include much noise, and
(b) images that have been photographed with spectral characteristics that severally include an invisible light or a different color not to be near to the target and include little noise.
Then, the technique utilizes the images having the plurality of different spectral characteristics to obtain an image having a correct color and little noise.
A configuration of noise reduction processing using a solid state imaging device of a single plate color system using the algorithm shown by the related art technique is described with reference to FIG. 4. The imaging device 30 is an imaging device of the Bayer color array of a general RGB array, which has been described above with reference to FIG. 1.
A plurality of images A and B is photographed using the imaging device 30, changing light sources. That is, the images A and B are as follows:
(the image A) an image that has been photographed with a spectral characteristic near to a target and include much noise, and
(the image B) an image that has been photographed with a spectral characteristic that includes an invisible light or a different color not to be near to the target and includes little noise.
The two mosaic images A and B are obtained by the photographing processing. The mosaic image A is dark and has much noise, but the color of which is correct. The mosaic image B is bright and includes little noise, but the color of which is incorrect. Two images of a high noise image A and a low noise image B are obtained by applying white balance processing of correcting pixel values to each spectrum to the photographed images A and B in white balance processing units 31a and 31b, respectively, and by executing demosaic processing to the images having been subjected to the white balance processing that has been described with reference to FIG. 3 in demosaic processing units 32a and 32b, respectively.
Moreover, the two images of the high noise image A and the low noise image B are input into a noise reduction processing unit 33, and the noise reduction processing based on these two images is executed. Then, an output RGB image is obtained.
The configuration and the processing of the noise reduction processing unit 33 are described with reference to FIG. 5. The noise reduction processing unit 33 uses the two images of the high noise image A and the low noise image B to produce the output RGB image, which has a color equal to the high noise image A with reduced noise by synthesizing these images.
An RGB low pass filter 42 is a cross bilateral filter, which is a kind of an edge preserving filter, and reduces the noise of the high noise image A with the edges of the image A, which has much noise, being preserved on the basis of the edges detected from the respective R, G and B components of the low noise image B as an input image.
An RGB low pass filter 41 applies a general FIR low pass filter, which is not especially equipped with any edge preserving functions, to each channel of R, G and B. An RGB high pass filter 45 obtains the high frequency component of each pixel value of R, G and B of the low noise image B, which is an input image having little noise. The acquisition of the high frequency components is performed by dividing the pixel values as the result of the application of the bilateral filter to the input images by the pixel values of the images before the application of the filter.
A blend executing unit 44 applies a previously set blend function to generate the image data having the pixel values obtained by multiplying the pixel values of an output image of the RGB low pass filter 42 by the pixel values of an output image of the RGB high pass filter 45, and to output the generated image data.
A speculum detecting unit 43 extracts the differences of the highlight, the shadow and the like in each pixel of the high noise image A and the low noise image B owing to the illumination change between the two images, and evaluates the height of the possibility of the occurrence of the differences of pixel values owing to the illumination change. A blend executing unit 46 performs the weighted addition of an output of the RGB low pass filter 41 and an output of the blend executing unit 44 on the basis of an evaluation result of the speculum detecting unit 43 to generate the output RGB image.
The blend executing unit 46 performs the setting of enlarging the weight of the pixel values output from the RGB low pass filter 41 in a pixel portion to which the speculum detecting unit 43 has judged that the possibility of the occurrence of the shadow or the highlight owing to the illumination change is high, and enlarging the weight of the pixel values output from the blend executing unit 44 in a pixel portion to which the speculum detecting unit 43 has judged that the possibility of the occurrence of the shadow or the highlight owing to the illumination change is low by regarding the pixel portion as the appearance of the detailed parts of a subject owing to the difference of illuminations. Thereby, the blend executing unit 46 performs the weighted addition of the output of the RGB low pass filter 41 and the output of the blend executing unit 44 to generate the output RGB image.
However, the algorithm is realized on the assumption that a plurality of images is photographed with different spectra, and consequently has a problem of the impossibility of applying the algorithm to a general digital camera and a movie camera, which can photograph an image only once at the same instant.